In recent years, multimedia that integrates audio, video and other information relating to the audio and the video has become popular. Conventional information media, namely, a newspaper, a magazine, a TV, a radio, and a telephone are integrated into the multimedia. In general, multimedia relates and represents not only letters but also graphics, voice and particularly an image. To make conventional information media be multimedia, it is essential to represent the information in a digital form.
When information quantity of the each information media is estimated, the information quantity per letter is 1˜2 bytes, while that of voice per second needs more than 64 kbits (quality of telephone conversation), and further, that of a moving picture per second needs more than 100 Mbits (standard TV broadcasting quality). Therefore, it is not realistic to utilize the enormous information of the information media as-is in the digital form. For example, a TV telephone has already been in practical use by ISDN (Integrated Service Digital Network) that has a transmission speed of 64 kbps˜1.5 Mbps, but it is impossible to transmit videos of a TV and a camera directly through ISDN.
Thus, technology for compressing information is necessary. For example, in the case of a TV telephone, the moving picture compression technology of H.261 or H.263 standard standardized internationally by ITU-T (International Telecommunication Union Telecommunication Standardization Section) is used. Additionally, it is possible by the MPEG-1 standard to contain image information together with voice information in an ordinary musical CD (Compact Disk).
Here, MPEG (Moving Picture Experts Group) is an international standard for digital compression of a moving picture signal and MPEG-1 is a standard for compressing the moving picture signal to 1.5 Mbps, namely, about one-tenth of the information of a TV signal. Additionally, the transmission speed whose subject is MPEG-1 standard is mainly limited to about 1.5 Mbps; by MPEG-2 standardized to satisfy further demand for high image quality, the moving picture signal is compressed to 2˜15 Mbps.
Further, in the present circumstances, by the working group (ISO/IEC JTC1/SC29/WG11) that has carried on the standardization of MPEG-1 and MPEG-2, MPEG-4 that has further higher compression ratio has been standardized. MPEG-4 not only makes it possible first to execute cording highly efficiently with a low bit rate, but also powerful error resistance technology is introduced. The technology can minimize subjective image quality degradation even if an error in a transmission channel occurs. Furthermore, ISO/IEC and ITU-T jointly carry on standardization activity of JVT (Joint Video Team) as a next generation screen coding method; at the present moment, what is called Joint Model 2 (JM2) is the latest.
Different from the conventional moving picture coding, JVT can select an arbitrary image (picture) out of plural images (pictures) as a preceding reference image. Here, a picture represents a frame or a field.
FIG. 1A is a diagram that explains the image coding that executes coding referring to a selected image out of plural reference images stored in a memory. FIG. 1B is a structure diagram that shows the structure of the memory that stores images.
As shown in FIG. 1B, the memory is made up of short term memory and long term memory. The short term memory memorizes several images coded/decoded immediately before and corresponds to reference images of so-called a P picture (Predictive Coded Picture) and a B picture (Bi-predictive Coded Picture) of MPEG-1 and MPEG-2. The long term memory is used to store an image signal longer than the short term memory.
Ordinarily, the short term memory is an FIFO (first in, first out) memory. In the case of storing an image over the upper limit of the memory, the oldest image in the short term memory is released and a new image is stored in the area. Consequently, ordinarily to refer to a reference image that has been released from the memory by the mechanism of FIFO, it is possible to refer to the reference image for a long time by moving the reference image in advance from the short term memory to the long term memory and storing the image in the long term memory. The long term memory is a method for clearly designating storage area; it is possible to refer to the picture stored in the area unless the same area is designated and overwritten.
FIG. 1A shows a predictive situation at the time of image coding; the image of the picture (frame) No. 2 refers to the image of the picture No. 0; and the image of the picture No. 1 refers to the image of the picture No. 0 or the picture No. 2. Similarly, the image of the picture No. 4 refers to the images of the picture Nos. 0 and 2; and the image of the picture No. 6 refers to the image of the picture No. 0. Further, the image of the picture No. 5 can refer to the images of the picture Nos. 0, 2, 4 and 6.
In FIG. 1A, the images of the picture Nos. 0, 6 and 12 are referred to after a relatively long time, while the images of the picture Nos. 2, 4 and 8 are referred to only by images a short time later. Consequently, as is shown in FIG. 1B, the memory area to store images is divided into the short term memory and the long term memory; and the images of the picture Nos. 0,6 and 12 can be stored in the memory that needs the long time storage.
Now, to use the memory shown in FIG. 1A effectively requires sophisticated memory management. Accordingly, a mechanism to control the memory is introduced in JVT.
Commands for controlling the memory are as follows:    1. The command for selecting images that can be referred to;    2. The command for releasing the memory area that stores the images that are useless as the reference images of the predictive coding in the short term memory; and    3. The command for moving the contents of the short term memory to the long term memory.
In coding and decoding an image, since the image that has a small prediction error in the unit of a block is selected out of the images that can be referred to, a signal that designates the reference images in the unit of a block is necessary. By electing in advance the images that can be referred to, it is possible to reduce the number of candidates of the reference images into an appropriate value and to cut down the number of bits of the reference image designation signal that is necessary in the unit of a block.
Additionally, in the case of moving images from the short term memory to the long term memory, since it is useless to store the images of the same contents in both of the short term memory and the long term memory, the images in the short term memory are released.
FIG. 2A and FIG. 2B are flowcharts that show a conventional image coding method and a conventional image decoding method.
FIG. 2A shows operations that an image coding device executes to release the memory area that stores the pictures that are never used as the reference images in the predictive coding. In FIG. 2A, for a start, the image coding device codes the inputted image (Step 100). After the coding, the image coding device examines whether there is an unnecessary area (the images that are not referred to in future coding) in the memory (Step 101) and judges whether a picture in the memory which is never used as reference exists or not (Step 102). When the picture in the memory which is never used as reference is judged to exist (Yes at Step 102), the command for releasing a picture in the memory which is never used as reference is coded as memory management information (Step 103), a picture in the memory which is never used as reference is released (an images in the memory are deleted) (Step 104) and the processing ends. On the other hand, in the case of the image coding device judging that there is no picture in the memory which is never used as reference (No at Step 102), the processing ends without executing the operations in Step 103 and Step 104.
Next, operations that an image decoding device executes to release the memory area that stores the images which are never used as reference in the predictive coding is explained based on the flowchart of FIG. 2B. For a start, the image decoding device decodes the memory management information (Step 110) and decodes an image signal from a coded signal (Step 111). The image decoding device judges whether a memory release command exists or not as a result of the examination (Step 112). If the memory release command exists (Yes at Step 112), the image decoding device judges whether there are images that should be released by the command or whether the memory has been already released (the images have already been deleted) (Step 113). If the image decoding device judges that the memory has already been released (Yes at Step 112), the image decoding device assumes that it is an ERROR because JVT prohibits transmitting the command for releasing the same image again after releasing the image from the memory and therefore to release again the memory that has already been released is the ERROR. On the other hand, if the image decoding device judges that the memory is not released (No at Step 113); it releases the memory (Step 114) and ends the processing. If it is judged that there is no memory release command (No at Step 112), the operations of Step 110 and Step 111 are not executed and the processing ends. By the way, Step 110 and Step 111 are in no particular order and may counterchange.
FIG. 3A and FIG. 3B are flowcharts that show another conventional image coding method and another conventional image decoding method.
FIG. 3A shows operations that the image coding device executes to move the images from the short term memory to the long term memory.
In FIG. 3A, for a start, the image coding device codes the inputted image (Step 120). After the coding, the image coding device examines whether there are images that should be moved to the long term memory (Step 121) and judges whether there are the images that should be moved or not (Step 122). If there are the images that should be moved (Yes at Step 122), the image coding device codes a command for indicating how the images are moved to the long term memory as the memory management information (Step 123), moves the images to the long term memory based on the command (Step 124) and ends the processing. On the other hand, in the case of the image coding device judging that there is no image that should be moved to the long term memory (No at Step 122), the image coding device does not execute the operations of Step 123 and Step 124 and ends the processing.
Next, the operations that the image decoding device executes to move the images from the short term memory to the long term memory are explained based on the flowchart of FIG. 3B. For a start, the image decoding device decodes the memory management information (Step 130). Next, the image decoding device decodes an image signal from a coded signal (Step 131). Then, the image decoding device judges whether there is a command for moving the images to the long term memory or not in the decoded memory management information (Step 132). If the image decoding device judges there is the command (Yes at Step 132), it judges next whether there are images that should be moved by the command or whether the images have already been moved (there is no image because the images are released after the move) (Step 133). JVT prohibits transmitting the command for moving the same image again to the long term memory after moving the image to the long term memory. Consequently, in the case of moving the image that has already been moved to the long term memory again to the long term memory, it is assumed to be the ERROR. Therefore, when the image decoding device judges that the image has already been moved (Yes at Step 133), the image decoding device assumes it to be the ERROR and when the image decoding device judges that the image is not moved, the image decoding device moves the image to the long term memory (Step 134) and ends the processing.
On the other hand, in the case of the image decoding device judging that there is no command for moving the image to the long term memory (No at Step 132), the image decoding device does not execute the operations of Step 133 and Step 134 and ends the processing. By the way, Step 130 and Step 131 are in no particular order and may counterchange.
FIG. 4A and FIG. 4B are flowcharts that show yet another conventional image coding method and yet another conventional image decoding method.
For a start, operations that the image coding device executes to select the image that can be referred to are explained based on the flowchart of FIG. 4A.
For a start, the image coding device selects reference images that are expected to have a high correlation with coded images (generally, reference images that are temporally close) as candidates of the reference images (Step 200). Next, the image coding device codes the designation information (a type of memory management information) that indicates the selected candidates of the reference images (Step 201), refers to and codes an appropriate reference image out of the selected candidates of the reference images in the unit of block (Step 202) and ends the processing. By the way, Step 201 and Step 202 are in no particular order and may counterchange.
Next, operations that the image decoding device executes to select the image that can be referred to are explained based on the flowchart of FIG. 4B.
For a start, the image decoding device decodes the designation information, a type of memory management information (Step 210), selects candidates of the reference images from the memory as a result (Step 211), selects an appropriate reference image out of the selected candidates of the reference images in the unit of block, decodes the appropriate reference image while referring to it (Step 212) and ends the processing.
Now, in the conventional image coding methods and the conventional image decoding methods, the image coding device codes and outputs the command for releasing images which are never used as reference from the memory and the command for moving images from the short term memory to the long term memory, and transmits the commands to the image decoding device, which decodes the commands. The number of this transmission is limited to only one picture and therefore in the case of the picture including the command being lost by a transmission error and the like, it is impossible to decode the image because the decoding device cannot decode the image placement in the memory correctly.
Furthermore, in coding and decoding an image, in the case of selecting a reference image, when the image coding device selects simply the temporally close images only as the reference image candidates, it is impossible to execute a most suitable coding considering the scalability (in an example of predictive structure in FIG. 1A, it is possible to decode an I picture (Intra Coded Picture) and a P picture (Predictive Coded Picture) even if a B picture (Bi-predictive Coded Picture) is not decoded or it is possible to decode other P pictures even if the P pictures of the picture Nos. 4, 10 and 16 are not decoded) for decoding an image. In other words, although the images that are temporally adjacent to the image of picture No. 6 are the images of picture Nos. 4 and 2, in reality, only the image of picture No. 0 can be referred to, and therefore, if the images of picture Nos. 4 and 2 are included in the candidates of the reference image, the efficiency for coding is not very good.
Further, in the conventional image coding methods, since it is prohibited to transmit the command for releasing the image which is never used as reference and the command for moving the images from the short term memory to the long term memory attaching to an image that is not stored in the memory, flexible transmission of the commands of memory management information is blocked. To prohibit transmitting the commands attaching to the image that is not stored in the memory has the following reason. Namely, the image that is not stored in the memory has the least significance; the possibility that it is not decoded by the scalability is high; the image decoding device cannot decode the image placement correctly because the commands attaching to the image that is not stored in this memory are not decoded; the reason to prohibit transmitting the commands attaching to the image that is not stored in the memory is to avoid the situation.